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There  are  considerable  on-going  efforts  within  the  Air  Force  Research  Laboratory  (AFRF)  to 
improve  the  science  and  technology  (S&T)  program.  Each  year  the  Air  Force  Scientific 
Advisory  Board  (SAB)  conducts  independent  quality  reviews  within  the  labs.  The  reviews 
assess  the  quality  and  long-term  relevance  of  the  Air  Force  S&T  program.  Quality  here  is 
broadly  defined  to  include  science,  people,  strategy,  resources,  focus,  facilities,  and  results. 
These  reviews  clearly  indicate  that  the  quality  of  the  S&T  program  is  improving  and  most  of  the 
projects  are  well  focused  on  future  Air  Force  capabilities. 

However,  based  on  extensive  benchmarking,  it  is  also  apparent  that  a  more  effective  S&T 
program  can  be  achieved.  This  study  offers  recommendations  that  apply  both  internally  and 
externally  to  AFRF.  The  members  of  this  study  team  strongly  urge  that  these  recommendations 
be  adopted. 
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U.S.AIR  FORCE 


Concerns  Leadership  Raised 
Surrounding  S&  T  Program 


■  Air  Force  topline  budget  pressure 

■  Lack  of  an  effective  methodology  for  valuing  and 
prioritizing  S&T  investments  in  warfighter  terms 

■  Lack  of  visibility  into  the  program 

■  Extent  to  which  external  S&T  sources  (defense, 
industry,  academia)  offset  the  need  for  Air  Force 
S&T  investment 

■  Efficiency  of  S&T  program  execution 
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Enormous  Air  Force  topline  budget  pressures  adversely  impact  modernization  and  infrastructure 
as  well  as  emerging  mission  area  resource  requirements.  The  S&T  program  has  taken  substantial 
budget  cuts  as  a  consequence. 

There  is  a  lack  of  an  effective  methodology  for  valuing  and  prioritizing  S&T  investments  in 
warfighter  terms.  Hence,  the  benefits  of  an  increased  (or  decreased)  S&T  investment  versus 
other  needs  are  unclear.  It  is  also  unclear  what  the  core  science  investment  contributes  to  the 
longer-term  needs  of  the  Air  Force.  To  some,  the  basic  and  longer-term  research  efforts  appear 
to  be  a  jobs  program. 

There  is  a  lack  of  senior  leadership  visibility  into  the  program.  There  are  many  projects 
spanning  a  wide  variety  of  needs  addressing  the  short,  intermediate,  and  long  term.  The  inherent 
complexity  makes  characterization  of  the  S&T  portfolio  a  significant  communication  challenge. 
Consequently,  there  has  not  been  a  shared  sense  of  direction  and  buy-in  at  the  executive  level. 

It  is  not  clear  to  what  extent  non- Air  Force  S&T  investments — for  example,  from  the  Defense 
Advanced  Research  Projects  Agency  (DARPA) — reduce  the  need  for  Air  Force  S&T  investment. 

Finally,  the  efficiency  of  execution  is  questioned.  Feadership  would  like  to  see  more  bang  for 
the  buck.  Many  perceive  that  not  enough  gets  out  of  the  laboratories  and  into  the  hands  of  the 
warfighter,  and  that  the  transition  process  takes  too  long. 
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Terms  of  Reference 


■  Tie  S&T  investment  strategy  to  the  Air  Force 
long-range  plan  and  vision 

■  Take  into  account  the  investment  of  others  in 
the  Air  Force  S&T  strategy,  but  recognize  the 
risk;  partnership  with  industry 

■  Recommend  industry  tools  (practices)  for  the 
Air  Force  to  use  for  determining  S&T 
investment 

■  Develop  augmenting  strategies  for  workforce 
management,  risk  assessment,  and  hedges 

■  Recommend  a  level  for  the  Air  Force  S&T 
topline 


Integrity  -  Service  ■  Excellence 


The  S&T  Vision  Study  was  charted  by  the  Secretary  of  the  Air  Force  (SecAF)  and  the  Chief  of 

Staff  of  the  Air  Force  (CSAF)  as  one  of  four  studies  for  2000.  The  Terms  of  Reference  provided 

the  following  charter,  and  are  based  on  the  concerns  previously  mentioned: 

1.  Develop  a  technology  investment  strategy  guided  by  the  Air  Force  Long  Range  Plans,  Air 
Force  Vision,  and  U.S.  National  and  Military  Strategies.  In  addition,  provide  technology 
possibilities  and/or  opportunities. 

2.  Review  industry  investments  to  determine  level  of  effort  in  Air  Force  S&T  being  mindful  of 
the  potential  divestiture  by  industry  in  any  given  area. 

3.  Review  the  tools  that  industry  uses  to  determine  where  to  invest,  form  strategic  partnerships, 
and  recommend  tools  that  can  by  used  by  AFRL. 

4.  Develop  and  recommend  strategies  for  S&T  investment  at  all  levels  to  include  industry 
partnerships,  workforce  implications,  risk  assessment,  and  hedges. 

5.  From  the  above,  recommend  a  level  for  the  Air  Force  S&T  topline  budget. 
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U.S.  AIR  FORCE 

Panel  Members 

Dr.  W.  Ballhaus,  Jr.,  Study  Chair 

Mr.  J.  Mattice 

Mr.  J.  Grant,  Study  Vice  Chair 

Dr.  L.  Metzger,  Advisor 

Dr.  R.  Barthelemy 

Lt  Gen  (R)  G.  Muellner,  Panel  Chair 

Col  (R)  W.A.  Byrne 

Lt  Gen  (R)  M.  O’Neill,  Ph.D.,  Panel  Chair 

Dr.  A.  Chester 

MGen  (R)  R.  Paul,  Advisor 

Mrs.  N.  Crawford,  Advisor 

Dr.  A.  Pensa,  Panel  Chair 

Col  D.  Erbschloe,  Ph.D. 

MGen  (R)  R.  Rankine,  Jr.,  Ph.D. 

Dr.  J.  Fender 

Dr.  E.  Reichmanis 

Dr.  M.  Ganz 

Dr.  K.  Richey 

Dr.  V.  Gawron,  Panel  Chair 

Dr.  W.  Rouse 

Lt  Col  W.  Hammond 

Dr.  R.  Selden,  Advisor 

Dr.  D.  Hastings,  Panel  Chair 

Dr.  G.  Yonas 

Dr.  J.  Hendler 

Maj  L.  Merkle,  Ph.D.,  Tech  Editor 

MGen  (R)  D.  Lamberson,  Ph.D. 

Ms.  M.  Darby,  Lead  Executive  Officer 

Ms.  T.  Lunt 

Lt  Col  P.  Schubert,  Executive  Officer 

Integrity  - S 

e  r  v  i  c  e  -  Excellence 

The  study  panel  consisted  of  five  retired  generals  (12  stars),  three  prior  S&T  laboratory 
commanders  (two  Air  Force  and  one  Army),  three  Air  Force  Chief  Scientists  (one  current  and 
two  former),  and  numerous  members  from  industry,  academia,  and  Federally  Funded  Research 
and  Development  Centers  with  substantial  S&T  experience. 

Panel  members  were  divided  into  five  teams.  Team  1  was  led  by  Lt  Gen  (R)  George  Muellner. 
Members  included  MGen  (R)  Bob  Rankine  and  Dr.  Gerold  Yonas.  The  team  addressed  the  role 
of  the  Chief  Executive  Officer  (CEO)  in  terms  of  guidance  and  participation  in  the  S&T 
program.  Team  2  was  chaired  by  Lt  Gen  (R)  Mai  O’Neill.  Members  included  Dr.  Art  Chester, 
Dr.  Matt  Ganz,  Dr.  Keith  Richey,  and  Dr.  Bill  Rouse.  This  team  focused  on  the  linkages  with 
other  organizations  and  the  problems  associated  with  the  transition  of  technology  from  the 
AFRL.  Team  3  was  overseen  by  Dr.  Tony  Pensa.  Members  included  Col  (R)  Bill  Byme,  Dr. 
Janet  Fender,  and  Mr.  Jim  Mattice.  The  team  evaluated  topline  funding  guidelines  and  improved 
ways  of  leveraging  external  sources.  Team  4  was  headed  by  Dr.  Valerie  Gawron.  Members 
were  Col  Don  Erbschloe,  Lt  Col  Walt  Hammond,  and  Ms.  Teresa  Lunt.  This  team  documented 
the  current  Air  Force  S&T  planning  process,  benchmarked  the  processes  of  comparable 
government  agencies  and  companies,  and  reviewed  studies  on  improving  planning  processes. 
Team  5  was  led  by  Dr.  Dan  Hastings.  Members  were  Dr.  Bart  Barthelemy,  Dr.  Jim  Hendler, 
MGen  (R)  Don  Lambcrson,  and  Dr.  Elsa  Reichmanis.  The  team  examined  people,  facility,  and 
organizational  issues. 

The  roles  of  the  remaining  panel  members  are  indicated  on  the  chart. 
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Terms  of  Reference 


Tie  S&T  investment  strategy  to  the 
Air  Force  long  range  plan  &  vision 

Take  into  account  the  investment 
of  others  in  the  Air  Force  S&T 
strategy,  but  recognize  the  risk; 
partnership  with  industry 

s 

Recommend  industry  tools 
(practices)  for  Air  Force  to  use  for 
determining  S&T  investment 

s 

Develop  augmenting  strategies  for 
workforce  management,  risk 
assessment,  and  hedges 

s 

Recommend  a  level  for  the  Air 

Force  S&T  topline 

Alternative:  process  for  valuing 

10/13/00 


Integrity  ■  Service  -  Excellence 


The  study  team  considered  itself  compliant  with  the  Terms  of  Reference  with  the  exception  of 
recommending  a  level  of  investment  for  the  Air  Force  S&T  topline  budget,  which  is  explained 
on  the  following  two  charts. 
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■  Previous  studies  have  established  a  comprehensive 
compilation  of  industrial  S&T  investment 

■  DSB  1998:  Mr.  Walter  E.  Morrow  Jr./MIT  Lincoln 
Laboratory 

■  Basic  research  (0.05%  to  3.5%) 

■  Development  focused  research  (0.05%  to  15%) 

■  No  unassailable  way  to  establish  topline  from 
these  data 
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Arbitrarily  setting  the  topline,  by  whatever  means,  has  a  number  of  drawbacks.  It  restricts  top 
leadership’s  ability  to  manage  the  total  budget  in  times  of  extreme  constraints.  It  does  not 
account  for  programmatic  changes  that  naturally  occur  from  year  to  year,  such  as  the 
cancellation  of  a  large  program  or  the  addition  of  a  major  new  activity.  Perhaps  most 
detrimental,  it  can  lead  to  an  entitlement  mentality  within  the  organization. 

In  1998,  a  Defense  Science  Board  (DSB)  study  Chaired  by  Walter  Morrow,  former  Director  of 
the  Massachusetts  Institute  of  Technology  (MIT)  Lincoln  Laboratory,  conducted  a 
comprehensive  look  at  S&T  spending  in  the  commercial  sector.  The  levels  referred  to  are  not 
easily  associated  with  the  Air  Force  S&T  levels.  In  all  cases,  the  study  found  that  the  amount  of 
investment  is  driven  by  the  core  business  technology  needs  and  is  determined  through  a  bottom- 
up  methodology.  The  commercial  sector  uses  the  technology  investment  “level”  as  a  guideline. 

It  provides  a  sanity  check  or  test  for  reasonableness. 
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(Cont’d) 
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■  Industrial  R&D  investment  determined  by  technology  needs  to 
support  core  businesses 

■  Budget  adjustment  consequences  defined  in  terms  of  business 
impact 

■  Level  of  investment  guidelines  used  after  the  fact  to  ensure 
“ballpark”  posture  relative  to  competition  and  historical  levels 

■  This  study  proposes  an  analogous  process 

■  Provides  Air  Force  leadership  with  insight  into  the  content  and 
value  of  the  S&T  program 

■  Defines  budget  adjustment  consequences  in  warfighter  terms 

■  Resulting  topline  should  be  tested  (sanity-checked)  against 
historical  levels  (1 .8-2.2%)  and  special  needs 
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Industrial  research  and  development  (R&D)  investment  is  determined  by  the  technology  needed 
to  support  the  business.  Although  the  process  varies  considerably  from  business  to  business,  it 
generally  contains  many  common  elements.  Leadership  provides  a  few  guidelines  that  are 
consistent  with  the  strategic  plan  and  take  into  consideration  extraordinary  investment  needs, 
such  as  a  new  product  line,  a  major  customer  need,  or  a  perceived  competitive  threat.  The 
organization  then  recommends  individual  R&D  projects  consistent  with  these  guidelines  and 
normal  business  needs.  Top  leadership  reviews  the  recommendations,  taking  into  consideration 
affordability  and  using  mle- of-  thumb  historical  investment  levels  as  a  sanity  check.  Several 
iterations  are  often  required  to  arrive  at  a  final  budget  and  a  list  of  approved  projects. 

This  study  will  propose  a  process  for  annually  establishing  the  S&T  topline  budget  based  on 
warfighter  needs  (both  present  and  future)  and  affordability.  This  allows  the  S&T  portfolio  to  be 
characterized  in  terms  of  warfighting  capabilities  and  the  S&T  core  competencies  required  to 
enable  those  capabilities.  The  S&T  investment  can  be  valued  against  other  compelling  Air  Force 
budget  needs.  At  the  end  of  this  process,  the  S&T  investment,  as  a  percent  of  Air  Force  total 
budget,  should  be  compared  with  historical  levels,  which  typically  are  in  the  range  1.8  percent  to 
2.2  percent.  This  is  analogous  to  the  industry  sanity  check,  and  if  the  S&T  investment  is  out  of 
this  range,  the  rationale  should  be  explainable  based  on  special  circumstances  or  needs. 


7 


Slide  8:  Air  Force  S&T — Why  Invest? 
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Air  Force  S&T 

Why  invest? 


■  Create  technology  options  in  time  to  meet  emergent 
warfighting  needs 

■  Shape  the  future  (game  changers — for  example, 
stealth,  GPS)  and  avoid  technological  surprises 

■  Position  U.S.  industry  to  deliver  and  sustain 
technologically  superior  systems 

■  Maintain  in-house  expertise  to  make  good 
technology  acquisition  decisions — be  a  smart  buyer 

■  React  rapidly  to  solve  the  Air  Force’s  technical 
problems  (911  service) 


Integrity  -  Service  -  Excellence 


The  yin  and  yang  of  a  successful  S&T  program  are  requirements  pull  and  technology  push — both 
are  essential.  The  portion  of  the  program  pulled  by  requirements  creates  technology  options  in 
time  to  meet  emergent  warfighter  needs,  such  as  the  engines  and  avionics  of  the  F-22.  But 
innovation  arising  from  technology  breakthroughs  often  creates  new,  previously  un imagined, 
system  concepts  that  can  reshape  future  Air  Force  operations.  Past  examples  of  these  “game 
changers”  include  stealth,  the  Global  Positioning  System  (GPS),  night  vision,  and  Joint 
Surveillance  Target  Attack  Radar  System,  all  of  which  contributed  markedly  to  the  successful 
outcome  in  the  Persian  Gulf  War.  Initially  these  new  concepts  were  reluctantly  embraced  by  the 
operational  community,  but  they  have  subsequently  become  essential  elements  of  Air  Force 
concept  of  operations  (CONOPS). 

Most  of  the  Air  Force  S&T  program  is  executed  under  contracts  with  industry  in  order  to  infuse 
new  technology  into  companies  that  will  eventually  deliver  and  sustain  the  new  weapon  systems. 
However,  some  portion  of  the  S&T  budget  must  be  directed  toward  AFRL  in-house  research  to 
connect  AFRL  scientists  and  engineers  to  the  global  research  community  so  they  can  select  the 
right  technologies  to  pursue  on  contract,  be  aware  of  the  state  of  the  technology  developments  of 
others,  and  rapidly  react  to  solve  immediate  Air  Force  technical  problems. 


It  is  important  to  recognize  that  industry  invests  in  S&T  in  an  area  unique  to  Air  Force  needs  in 
response  to  Air  Force  investments  in  that  area.  Industry  uses  such  Air  Force  investment  as  an 
indicator  that  the  Air  Force  is  serious  about  the  development  of  that  technology  and  its  use  in 
potential  future  systems. 
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Slide  9:  Air  Force  and  Industry  S&T — Key  Contrasts 
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prioritizes  S&T  investments 

prioritizes  S&T 
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“RESULTS”  align 

Integrity  -  S  e 

r  v  i  c  e  -  Excellence 

Industry  S&T  is  driven  by  the  need  to  meet  near-term  financial  objectives  in  the  marketplace, 
and  industry  investment  is  driven  by  the  business  case.  Air  Force  S&T  is  driven  by  the  need  to 
meet  warfighter  needs  in  the  battlespace.  As  a  result,  industry  prioritizes  its  S&T  investments  to 
capture  near-term,  high-return,  and  high- dollar- value  programs.  The  Air  Force,  on  the  other 
hand,  must  prioritize  its  S&T  investments  to  ensure  that  it  has  technically  superior  weapons  and 
support  systems  that  will  enable  continuing  mastery  of  the  aerospace  domain.  It  is  only  when 
these  industry  and  Air  Force  priorities  coincide  that  the  Air  Force  can  depend  upon  industry  S&T 
investments  to  address  Air  Force  needs.  Generally,  industry  invests  when 

•  A  major  system  procurement  is  anticipated 

•  A  broader  market  can  be  pursued 

•  The  Air  Force  invests  in  S&T,  leading  to  a  future  system  acquisition 

Anticipation  of  a  major  system  procurement  will  drive  industry  to  develop  proprietary 
technology  as  a  discriminator,  but  generally  for  near-term  applications  only.  Long-term,  Air 
Force-unique  S&T  is  usually  only  accomplished  by  Air  Force  funding — either  contracted  or  via 
an  in-house  effort. 
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Slide  10:  Organizations  Reviewed 
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Integrity  -  Service  ■  Excellence 


Panel  members  visited,  consulted  with,  or  were  briefed  by  numerous  companies  and 
organizations  in  industry,  the  Air  Force,  and  other  government  agencies. 

The  panel  focused  on  two  primary  technological  areas  in  its  sampling  from  industry — aerospace 
and  information  technology.  In  particular,  common  attributes  and  best  practices  were  sought. 
Eight  companies  were  selected  on  the  basis  of  excellence  in  their  respective  markets  as  well  as 
in-depth  familiarity  by  the  study  panel  members.  In  addition,  the  panel  reviewed  three 
companies  (3M,  DuPont,  and  Nokia)  that  were  part  of  a  recent  global  benchmarking  study 
conducted  by  Professor  Ed  Roberts  of  the  MIT  Sloan  School. 

The  Air  Force  organizations  represent  the  key  players  and  stakeholders  in  the  Air  Force  S&T 
enterprise,  including  the  planners  (Air  Staff),  executors  (AFRF),  and  customers  (product  centers 
and  major  commands  [MAJCOMs]). 

Finally,  the  Air  Force  S&T  process  was  compared  to  and  contrasted  against  those  from  other 
Services,  Department  of  Defense  (DoD)  organizations,  and  government  agencies,  including  the 
Defence  Evaluation  and  Research  Agency  (DERA)  of  the  U.K. 
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Slide  11:  Top-Level  Findings 


This  simplified  diagram  depicts  an  S&T  process  extracted  from  companies  with  the  best 
practices.  The  top  leadership  drives  the  process  by  articulating  a  clear  vision  accompanied  by  a 
few  specific  goals  that  are  both  an  impetus  to  action  and  collectively  drive  the  enterprise  to  a 
desired  future  state. 

The  strategic  plan,  customer  needs  (warfighter),  and  technology  opportunities,  in  turn,  drive  S&T 
planning,  which  culminates  in  an  S&T  plan.  Customer  needs  represent  the  user  pull,  and 
technology  opportunities  represent  the  technology  push.  It  is  critical  that  there  is  a  reasonable 
balance  between  these  two.  The  S&T  plan  consists  of  the  individual  projects  and  performance 
objectives  clearly  linked  to  major  goals  and  critical  future  capabilities  (CFCs).  In  addition,  the 
plan  addresses  cost  and  schedule.  The  plan  and  the  budget  realities  need  to  be  rationalized 
through  an  iterative  process  that  is  depicted  by  the  feedback  arrow  between  “S&T  Budget”  and 
“S&T  Planning.”  The  plan  drives  the  execution  phase.  If  cost,  schedule,  and  technical 
performance  baselines  are  established  for  S&T  deliverables  (primarily  in  the  6.3  portion  of  the 
portfolio),  then  those  responsible  for  execution  can  more  easily  be  held  accountable. 

Relative  to  this  process,  the  SAB  study  identified  three  major  areas  (leadership,  planning,  and 
execution)  where  improvements  would  enhance  the  effectiveness  of  the  overall  S&T  program. 
Leadership  issues  involve  goal  setting,  advocacy,  and  accountability.  Planning  concerns  relate  to 
linking  user  requirements  to  individual  technology  projects,  the  transition  from  6.3  to  6.4,  and 
the  inability  to  value  the  S&T  portfolio.  Finally,  execution  concerns  center  around  leveraging 
external  resources  and  maintaining  a  viable  science  and  engineering  (S&E)  workforce. 
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Slide  12:  Attributes  of  Successful  Enterprises — Benchmarking  Results 
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Attributes  of  Successful 

Enterprises 


Benchmarking  Results 

■  Leadership  involvement:  The  CEO  champions  the  vision 
and  top-level  strategy  and  goals  and  recognizes  S&T  as 
important  to  both  present  and  future  business 

■  Vision:  A  widely  shared  and  clear  idea  of  purpose  that  is  the 
basis  for  action  throughout  the  corporation 

■  Strategy:  A  shared  sense  of  direction — how  to  achieve  the 
vision 

■  Goals:  A  few,  clear,  compelling,  measurable  objectives, 
derived  from  the  vision,  supporting  the  strategy,  that  stimulate 
S&T  planning  and  progress 
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Integrity  ■  Service  -  Excellence 


Successful  corporations  are  driven  by  a  clear  understanding  of  their  vision.  In  Built  to  Last: 
Successful  Habits  of  Visionary  Companies,  James  C.  Collins  and  Jerry  I.  Porras  describes  the 
habits  of  visionary  companies,  “they  preserve  a  cherished  core  ideology  while  simultaneously 
stimulating  progress  and  change  in  everything  that  is  not  part  of  their  core  ideology. . .  the  vision 
builds  on  the  interplay  between  what  we  stand  for  and  why  we  exist  that  does  not  change  and 
sets  forth  what  we  aspire  to  become,  to  achieve,  to  create  that  will  require  significant  change  and 
progress.” 

The  vision  is  championed  by  the  CEO  and  is  widely  shared  throughout  all  levels  of  the 
organization.  The  vision  provides  the  long-term  strategic  direction  and  includes  a  few  critical 
goals  without  which  the  vision  could  not  be  achieved.  Clear,  compelling,  measurable  goals  unite 
and  drive  the  organization  to  achieve  critical  milestones  on  schedule  and  within  budget.  A  goal 
should  be  challenging,  exciting,  highly  motivating,  easily  communicated,  and  substantially 
achievable  within  a  given  timeframe. 
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Slide  13:  Attributes  of  Successful  Enterprises — Benchmarking  Results  (continued) 
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Attributes  of  Successful 

Enterprises 


Benchmarking  Results  (cont’d) 

■  Plan:  The  goals  allow  the  creation  of  an  S&T  plan  that  provides 
coherent  schedules  and  investments  for  attaining  the  goals 

■  Provides  clear  linkages  between  customer  needs  and  S&T  projects 

■  Employs  tools  for  valuing  and  prioritizing  investment  opportunities 

■  Emphasizes  buy  before  make — high  awareness  of  relevant 
marketplace 

■  Accountability:  The  CEO  holds  the  CTO  accountable  for 
executing  the  S&T  plans  to  achieve  the  goals 

■  Motivated  workforce:  Quality  people  driven  by  culture  of 
excellence 
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Integrity  ■  Service  -  Excellence 


The  goals  are  a  key  input  into  the  S&T  planning  process,  although  many  other  needs  that  must  be 
addressed  as  well.  The  plan  provides  clear  linkages  between  customer  needs  and  the  S&T 
projects.  Even  though  the  specific  plans  can  cover  extended  periods  of  time,  the  goals  and  the 
progress  toward  those  goals  are  reevaluated  and  reaffirmed  or  modified  every  year  to  adjust  for 
changes  in  markets,  competitive  forces,  suppliers,  and  relevant  technology.  Tools  or 
methodologies  are  used  to  value  and  prioritize  investment  opportunities.  Competitive  pressures 
force  the  organization  to  avoid  reinventing  the  wheel.  As  a  result,  the  culture  encourages  “buy 
before  make.” 

The  CEO  holds  the  chief  technology  officer  (CTO)  accountable  for  achieving  definitive 
milestones  associated  with  the  goals,  including  cost,  schedule,  and  performance.  In  the  absence 
of  a  few  goals,  it  is  more  difficult  to  hold  the  CTO  accountable  due  to  the  sheer  quantity  of 
disparate  projects  in  a  typical  large  S&T  program. 

Any  S&T  organization  rises  and  falls  with  the  quality  of  its  people.  While  excellent  facilities  are 
important,  it  is  people,  either  in  teams  or  as  individuals,  who  accomplish  the  mission.  The 
people  are  supported  by  and  supportive  of  a  culture  of  excellence.  This  means  that  poor-quality 
work  is  not  considered  acceptable  and  is  quickly  expunged.  It  means  that  respect  flows  from  the 
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quality  of  ideas,  depth  of  analysis,  innovation,  and  leadership  shown  by  individuals  in  the 
laboratory.  This  culture  of  excellence  is  one  that  is  never  satisfied  with  the  status  quo,  but  is 
always  interested  in  doing  better,  every  year,  by  all  metrics  of  importance  in  the  organization. 
This  culture  of  excellence  is  always  pushing  the  people  and  organization  to  be  “world  class”  in 
their  work  or  to  leave  areas  where  they  cannot  achieve  this  stature. 
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Slide  14:  Leadership 


w 


U.S.  AIR  FORCE 


Outline 


/leadership  VisioiL 
&  Guidance 


-(^ccountabjuty) 


Customer 
Needs 
(Pull) 


/Strategy), 


'echnology' 
^Opportunities^ 
'Push) 


S&T  Planning 


1 

i 

V 

(^S&T  Budget^ 

1 

f 

<)^Execution^ 

Leadership 

■  Goals 

■  Advocacy 

■  Accountability 

Planning 

■  Development  Planning 

■  Technology  Transitions 

■  Portfolio  Characterization 
Execution 

■  Leveraging  Partnerships 

■  Leveraging  Commercial 
Technology 

■  Workforce  Issues 


10/13/00 


Integrity  -  Service  -  Excellence 


15 


(This  Page  Intentionally  Left  Blank) 


16 


Goal  Setting 
Slide  15:  Findings 
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■  Fourteen  Critical  Future  Capabilities  (CFCs)  appear  in 
Volume  3  of  the  Air  Force  Strategic  Plan 

■  Comprehensively  express  needed  Air  Force  responses  to  the 
“compelling  demands  of  the  future  security  environment” 

■  However,  not  specific  enough  to  define  a  system  concept 
leading  to  a  technology  plan 

■  Critical  Future  Goals  (CFGs)  are  needed  that 

■  Are  derived  from  the  CFCs 

■  Are  explicit  enough  to  be  measured 

■  Are  problems  that  evoke  system  concept  solutions 

■  Move  the  Air  Force  toward  its  vision 


Top  leadership  defines  the  few  CFGs 
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Integrity  -  Service  -  Excellence 


The  14  CFCs  in  Volume  3  of  the  Air  Force  Strategic  Plan  provide  a  comprehensive  framework 
for  expressing  needed  Air  Force  responses  to  the  “compelling  demands  of  the  future  security 
environment.”  However,  they  are  not  sufficiently  specific  to  define  a  system  concept  leading  to 
a  technology  plan.  In  fact,  each  could  harbor  a  variety  of  system  concepts. 

Critical  Future  Goals  (CFGs)  are  needed  that  represent  the  top  leadership’s  highest  priorities. 

They  should  be  expressed  in  a  way  that  describes  a  needed  capability  and  evokes  system  concept 
solutions.  They  should  be  derived  from  the  CFCs  and  be  explicit  enough  to  be  measured.  There 
may  be  more  than  one  CFG  per  CFC,  but  there  also  may  be  CFCs  without  CFGs.  The  important 
point  is  that  the  group  of  CFGs  is  championed  by  the  leadership.  The  CFGs  collectively  describe 
the  desired  future  state  of  the  Air  Force,  and  they  define  a  timetable  for  achieving  that  future 
state.  They  make  the  vision  real,  understandable,  and  actionable.  For  example,  finding  targets 
under  trees  might  be  classed  as  a  CFG.  It  responds  in  part  to  the  CFC  referred  to  as  Precision 
Engagement — “Create  precise  effects  rapidly,  with  the  ability  to  retarget  quickly,  against  large 
target  sets  anywhere,  anytime,  for  as  long  as  required.”  Of  course,  a  great  deal  more  specificity 
would  be  required  to  make  this  example  actionable. 
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Slide  16:  S&T  Linkage  to  the  Vision 
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The  Air  Force  Strategic  Plan,  Vol.  3,  Provides  a 
u.s.  air  force  Characterization  for  Formalizing  Requirements  Pull  for  S&T 

Today  OPR  Desired 
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The  existing  linkage  between  vision  and  enabling  technologies  is  missing  several  important 
steps.  CFCs  are  well  stated  and  by  intent  cover  a  very  broad  spectrum. 1  However,  it  is  very 
difficult  for  the  AFRL  leadership  to  construct  a  prioritized  S&T  plan  from  these  broad 
descriptions.  Stated  differently,  almost  any  conceived  S&T  project  can  be  linked  to  a  CFC. 

Clearly  stated  CFGs  would  enable  top  leadership  to  express  areas  of  particular  importance  and 
bring  more  focus  to  the  process.  CONOPS  and  associated  system  concepts  would  be  developed 
by  operators  and  development  planners  (to  be  discussed  later).  These  concepts,  in  turn,  would 
drive  S&T  execution  plans  to  achieve  the  system  concepts  in  a  timely  fashion.  Hence,  enabling 
technologies  could  be  readily  identified. 

It  is  important  to  note  that  not  all  enabling  technologies  would  be  generated  in  this  manner,  only 
those  that  can  be  direcdy  linked  to  CFGs.  Other  customer  needs  and  technology  push  would 
evolve  the  balance. 


Air  Force  CONOPS  2020  provides  the  vision  by  articulating,  in  six  mission  areas,  how  aerospace  power  is 
executed.  The  14  CFCs  have  been  mapped  to  these  six  mission  areas. 
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Slide  17:  Linkage  Example — Targets  Under  Trees 
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Find,  identify,  and  kill  TUT  within  3  hours  of 
tasking  on  a  global  basis 
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Preparation  of  Battlefield 

Near-term  TUT  Technology  Demo  by  FY03  for 
$150  million 
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This  example  describes  a  3- year  S&T  plan  to  create  and  demonstrate  the  capability  to  find, 
identify,  and  kill  TUT.  It  demands  focused  investments  in  specific  technical  areas  involving 
sensors,  information  technology,  and  advanced  weapons.  This  achievement  will  make  demands 
on  many  Ah  Force  S&T  and  operational  organizations  and  lead  to  a  fundamental  change  in 
warfighting  tactics  and  strategy.  This  goal,  as  part  of  an  overall  plan  to  deal  with  time- urgent 
relocatable  targets,  will  motivate  and  stimulate  a  substantial  portion  of  the  Ah  Force  S&T 
community  and  demand  the  integration  of  ah,  space,  and  information  operations.  This  example 
is  challenging,  exciting,  highly  motivating,  easily  communicated,  and  represents  a  significant 
goal  achievable  within  a  given  timeframe. 
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Slide  18:  Recommendations 
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■  Establish  or  reaffirm  CFGs  annually 
(SecAF,  CSAF,  and  MAJCOM/CCs) 

■  Establish  concept  to  achieve  each  CFG 
(Development  Planning  activity,  addressed  later) 

■  Create  an  S&T  execution  plan  for  each  CFG 
(AFRL/CC) 

■  Assign  a  program  manager  with  budget  authority  for 
each  CFG 

■  Present  execution  plans  for  approval  by  SecAF, 
CSAF,  and  MAJCOM/CCs  at  annual  review  of  entire 
S&T  portfolio 


Integrity  -  Service  -  Excellence 


The  CFGs  are  established  or  reaffirmed  by  the  top  leadership  (SecAF,  CSAF,  and 
MAJCOM/CCs)  who  spend  quality  time  pondering,  debating,  and  prioritizing  critical  future 
needs  and  capabilities  within  the  Air  Force.  This  is  not  envisioned  as  a  staff  job,  although  some 
staff  preparation  may  be  necessary. 

A  Development  Planning  organization,  to  be  addressed  later,  establishes  concepts  to  achieve 
each  CFG  by  working  closely  with  warfighters,  product  centers,  and  AFRL.  AFRL/CC  assigns  a 
program  manager  with  budget  authority  for  each  CFG.  The  program  manager  develops  a 
detailed  execution  plan,  including  a  roadmap  that  defines  the  end  state  as  well  as  intermediate 
milestones  and  deliverables  along  the  way.  Enabling  technologies  are  clearly  linked  to  the 
milestones  so  that  the  S&T  community  not  only  understands  the  performance  objectives,  but  also 
the  time- critical  nature  of  each  S&T  project.  These  plans  are  presented  for  approval  at  an  annual 
review  of  the  entire  S&T  portfolio  by  SecAF,  CSAF,  and  MAJCOM/CCs. 
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■  Lack  of  MAJCOM  sponsorship  reduces  the 
competitiveness  of  the  S&T  portfolio  in  the  budget 
process 
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In  these  times  of  constrained  budgets,  S&T  funding  reductions  are  difficult  to  avoid.  The  S&T 
program  element  has  taken  its  share  of  cuts;  some  would  argue  excessively  so.  The  issue  here  is 
not  so  much  the  cuts,  but  the  lack  of  representation  during  the  budget- cutting  process.  S&T 
needs  a  voice  like  other  budget  elements.  Top  leadership  should  fully  understand  the 
implications  of  S&T  budget  cuts  before  making  the  final  decision.  When  a  group  of  senior 
leaders  meets  to  balance  the  budget,  it  is  always  easier  to  cut  someone  else’s  budget  and  even 
easier  to  cut  the  budget  of  someone  who  is  not  represented. 

The  4- star  sponsorship  can  bring  the  perspective  and  vision  that  is  needed  to  close  the  gap 
between  those  in  the  S&T  community  who  can  envision  future  capabilities  based  on  evolving 
technology  and  the  warfighters  who  understand  operations  but  lack  the  technical  background  to 
realistically  project  future  opportunities  based  on  new  technologies. 


21 


Slide  20:  Recommendations 


W  S&T  A  dvocacy 

*  Recommendations 

U.S.  AIR  FORCE 


■  Advocate  and  defend  S&T  budget  to  include  both 
portfolio  content  and  infrastructure  roles  (AFMC/CC) 
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The  Assistant  Secretary  of  the  Air  Force,  Acquisition  (SAF/AQ)  advocates  and  defends  the  S&T 
budget  in  the  Pentagon.  In  addition,  uniformed  4- star  advocacy  for  the  S&T  program  and  budget 
is  essential.  Air  Force  Materiel  Command,  Commander  (AFMC/CC)  should  provide  this 
advocacy. 
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Slide  21:  Finding 
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Accountability 

Finding 


■  There  is  no  apparent  agreed-upon  baseline  in  the 
plan  for  cost,  schedule,  and  technical  performance 
for  most  S&T  program  deliverables 

■  Hence,  it  is  difficult  to  hold  the  AFRL/CC 
accountable 
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There  are  thousands  of  S&T  projects  each  with  cost,  schedule,  and  technical  performance 
requirements.  Many  of  these  project  plans  change  during  the  course  of  a  year  for  a  variety  of 
reasons.  There  is  no  apparent  agreed-upon  baseline  at  the  beginning  of  the  fiscal  year,  and 
certainly  the  baseline  changes  during  the  year. 

At  present,  there  is  no  two-way  accountability  involving  both  the  deliverables  on  cost  and 
schedule  by  the  AFMC/CC  and  the  provisions  of  adequate  resources  by  the  leadership.  The 
annual  review  provides  the  opportunity  to  reaffirm  the  vision,  strategy,  and  goals  and  to  share 
information  on  discoveries  and  assess  the  progress  toward  the  goals.  It  is  vital  that  adequate 
resources  are  provided  with  sufficient  predictability  to  deliver  results  without  inefficient  and 
demotivating  reprogramming,  rescheduling,  and  rebudgeting. 

Although  the  AFRL/CC  can  be  held  accountable  by  individual  customers,  it  is  difficult  to  hold 
him  accountable  for  the  total  S&T  program.  Establishing  specific  goals  would  bring  more  focus, 
alignment,  and  a  sense  of  priority  to  the  S&T  program,  making  it  easier  to  hold  AFRL/CC 
accountable  for  delivering  what  is  most  important  as  defined  by  the  Air  Force  leadership.  In 
addition,  more  visibility  into  the  S&T  program  would  enhance  accountability  and  buy-in  by  the 
leadership. 


23 


Slide  22:  Recommendations 


W  Accountability 

*  Recommendations 

U.S.  AIR  FORCE 


■  Report  regularly  on  execution  status  of  agreed-upon  S&T 
plan  (AFRL/CC) 

■  Hold  AFRL/CC  accountable  for  cost,  schedule,  and 
technical  performance  (Air  Force  SAE) 

■  Hold  program  managers  accountable  for  cost,  schedule, 
and  performance  for  each  CFG  (AFRL/CC) 

■  Establish  annual  S&T  program  review  (SecAF  and  CSAF) 

■  Attendees  similar  to  forum  in  which  commitments  are  made 

■  Development  Planner  reaffirms  technology  needed  for  concept 

■  Warfighter  reaffirms  requirement  and  CONOPS  for  concept 


Integrity  -  Service  -  Excellence 


The  AFRL/CC  has  the  responsibility  not  only  to  lead  the  execution  process,  but  also  to  represent 
the  status  of  the  programs  to  the  Air  Force  leadership.  He  must  also  maintain  a  commitment  to 
excellence  by  creating  and  sustaining  a  culture  of  innovation  that  is  vital  to  future  Air  Force 
success  in  anticipating  and  dealing  with  emerging  asymmetric  threats.  As  stated  in 
Joint  Vision  2020,  “An  experimentation  process  with  a  low  tolerance  for  error  makes  it  unlikely 
that  the  force  will  identify  and  nurture  the  most  relevant  and  productive  aspects  of  new  concepts, 
capabilities,  and  technology.”  The  annual  review  should  seek  to  stimulate  not  only  commitment 
and  accountability  but  also  the  flexibility  to  make  wise  decisions  in  the  face  of  uncertainty  that 
permeates  fields  with  rapidly  changing  technology. 
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Development  Planning 
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■  Air  Force  has  no  systematic  process  for 

■  Evaluating  system  concepts  and  CONOPS  which 
leverage  technology  across  organizational  boundaries 
(SPOs,  product  centers,  MAJCOMs) 

■  Connecting  operational  requirements  through 
concepts  to  technology 

■  Aggregating  and  prioritizing  S&T  investments 

■  Development  Planning  activity  has  atrophied 

■  There  is  a  need  to  establish  an  updated 
Development  Planning  function  within  the  Air  Force 
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The  Air  Force  lacks  a  systematic  process  for  development  planning.  Specifically,  there  are  no 
apparent  means  to  evaluate  cross-organization  concepts  and  CONOPS,  link  operational 
requirements  and  concepts  with  technology,  perform  system  trade  studies,  and  prioritize  S&T 
investments.  For  example,  finding  targets  under  trees  could  conceivably  be  accomplished  by 
either  an  airborne  platform  or  a  satellite  system.  Rigorous  trade  studies  need  to  be  conducted, 
free  from  organizational  bias,  to  arrive  at  an  optimum  solution. 

The  development  planning  activities  formerly  at  the  product  centers  have  atrophied  due  to 
reorganization  and  lack  of  funding.  The  current  Technology  Planning  Integrated  Product  Team 
process  has  not  been  directed  toward  focusing  S&T  investments  through  evaluation  of  future 
system  alternatives.  The  Air  Force  needs  to  take  lessons  learned  from  these  earlier  and  ongoing 
activities  and  establish  an  updated  development  planning  function  in  which  operators, 
developers,  and  technologists  working  together  can  rigorously  work  the  trade  space.  The  results 
of  this  process  then  provide  a  framework  for  setting  priorities  in  the  S&T  program. 
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■  Direct  establishment  of  an  institutional  capability  for 
Development  Planning  (SecAF  and  CSAF) 

■  AFMC/CC  provides  lead  for  Air  Force  and  ensures  bridge 
between  user  requirements  and  technology  evolution 

■  AFMC/CC  ensures  system-of-systems  focus  across 
product  lines 

■  Use  “mixing  bowl”  process  (ref:  SAB  Battlelab  Study)  to 
identify  and  prioritize  technology  for  new  concepts 

■  Establish  program  element  to  fund 

■  Use  results  of  the  Development  Planning  trade  studies 
to  focus  and  prioritize  S&T  investments  to  achieve 
CFCs  (AFRL/CC) 
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The  study  panel  recommends  that  the  SecAF  and  CSAF  direct  the  creation  of  an  institutionalized 
capability  for  development  planning  and  establish  a  program  element  to  fund  it.  AFMC/CC 
should  provide  the  lead  for  the  Air  Force  and  ensure  a  system-of-systems  focus  across  product 
lines.  In  addition,  AFMC/CC  (through  its  product  centers)  should  ensure  a  bridge  between  user 
requirements  and  technology  evolution.  The  mixing  bowl  process  can  be  employed  to  provide  a 
voice  for  all  stakeholders. 

The  panel  believes  that  the  Army  Materiel  Systems  Analysis  Activity  (AMSAA),  a  Field 
Operating  Agency  that  has  been  used  successfully  for  Army  development  planning,  provides  a 
candidate  organizational  model  for  Air  Force  development  planning.  However,  the  proposed  Air 
Force  activity  should  have  a  narrower  focus  than  AMSAA  and  require  strong  coupling  to  product 
centers  for  product- specific  systems  engineering  expertise.  The  panel  envisions  a  small 
organization  of  highly  competent  people  that  draws  on  the  resources  of  the  product  centers, 
MAJCOMS,  and  AFRL,  and  has  senior  leadership  visibility.  For  this  activity  to  have  leverage 
and  impact,  it  must  not  become  a  bureaucratic,  staff-level  exercise ! 

The  development  planning  function  should  deliver  options  for  consideration  and  enable  the 
AFRL/CC  to  prioritize  S&T  investments  to  effectively  support  Air  Force  CFCs  as  detailed  in 
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Volume  3  of  the  Air  Force  Strategic  Plan.  Functions  performed  by  development  planning  should 
include: 

•  Assisting  the  warfighter  in  quantifying  and  refining  operational  capabilities 

•  Synthesizing  and  analyzing  alternative  concepts  to  satisfy  needs 

•  Helping  AFRL  identify  enabling  S&T  initiatives,  impacts,  and  value 

•  Dealing  with  complexities  at  the  system-of-systems -level 

•  Quantifying  the  value  and  total  ownership  cost  of  options 
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This  chart  shows  a  conceptual  Air  Force  implementation  of  an  AMSAA-like  activity  for 
development  planning.  The  Development  Planning  Activity  reports  to  AFMC  and  provides  the 
Commander  with  an  effective  way  to  compare  alternate  concepts  to  meet  a  warfighter  need  or 
expanded  CONOPS.  For  example,  the  requirement  for  finding  and  targeting  tanks  under  trees 
might  be  addressed  with  space  surveillance,  helicopters,  unmanned  aerial  vehicles  (UAVs),  or 
other  means.  The  activity  would  act  as  an  objective  evaluator  to  advise  the  Commander  on  the 
benefits  and  risks  of  these  competing  systems’  concepts. 

The  Development  Planning  Activity  should  be  organized  in  direct  congruence  with  CFCs  in 
order  to  focus  on  crosscutting  technologies  and  CONOPS  and  systems -of- systems  solutions 
rather  than  product- specific  alternatives.  The  Development  Planning  Activity  should  participate 
with  the  Battlelabs  and  Air  Force  systems  analysis  activities  to  expand  the  applicability  of 
concept  studies  and  drive  future  CONOPS. 

The  need  for  detailed  engineering  insights  into  product  concepts  requires  a  strong  coupling 
within  AFMC  among  AFRL,  product  centers,  and  the  Development  Planning  Activity.  The 
product  centers  must  provide  the  necessary  systems  engineering  insight  while  AFRL  provides 
the  technological  possibilities.  It  is  especially  important  that  AFRL  have  a  keen  awareness  of 
relevant  commercial  technology  opportunities  as  well. 
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■  Organizational  and  budgetary  difficulties  hinder 
transitions  from  6.3  to  6.4 

■  Operator,  developer,  and  technologist  are 
organizationally  separated,  hindering  the  planning 
and  execution  of  technology  transition 

■  6.3  and  6.4  funding  streams  are  decoupled 

■  Current  Applied  Technology  Council  (ATC)  process 
shows  promise 

■  Includes  warfighter,  developer,  and  AFRL 

■  Provides  decision  forum  for  commitment  to  transition 
from  6.3  to  6.4 
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The  Panel  examined  the  transition  of  technology  into  operational  systems.  The  Panel  found 
organizational  and  budgetary  barriers  that  impede  transition  from  6.3  to  6.4.  The  operator, 
developer,  and  technologist  are  organizationally  separated,  adversely  impacting  the  planning  and 
execution.  Funding  pathways  are  similarly  isolated. 

Also,  concern  has  been  expressed  that  the  transition  to  6.4  signals  initiation  of  a  major  new 
system  start,  entry  into  the  Defense  Acquisition  Board  process,  etc.,  which  is  not  the  case.  In 
fact,  the  transition  process  begins  long  before  this  (while  concepts  are  being  evaluated),  and  acts 
to  stimulate  S&T  evolution  rather  than  commit  the  Air  Force  to  a  specific  system. 

The  Panel  was  favorably  impressed  with  the  relatively  new  Applied  Technology  Council  (ATC) 
process,  which  has  been  used  at  least  once  by  every  Product  Center  to  review  maturing 
technology.  The  ATC  brings  together  warfighter,  developer,  and  the  AFRL  and  provides  a 
decision  forum  where  all  parties  can  assess  and  commit  to  the  6.4  transition  or  terminate  lower- 
priority  6.3  programs  in  favor  of  ones  more  likely  to  transition. 


31 


Slide  28:  Recommendations 


W 


Transitions 


Recommendations 


U.S.  AIR  FORCE 


■  Expand  the  role  of  ATC  (AFMC/CC,  operational 
MAJCOM/CCs,  and  SAF/AQ) 

■  AFMC/CC  chairs  ATC  when  competing  concepts 
involve  multiple  product  centers 

■  ATC  provides  a  forum  for  selection  and  prioritization 
of  concepts  for  system  development 

■  ATC  ensures  S&T  program  linkage  to  operational 
concepts 

■  Development  Planner  baselines  and  maintains 
transition  plans  for  ATC 
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The  Panel  recommends  unifying  and  expanding  the  role  of  the  ATCs,  which  should  provide  a 
forum  for  selection  and  prioritization  of  concepts  for  system  development  that  result  from  the 
development  planning  process  described  earlier.  When  competing  concepts  involve  multiple 
Product  Centers,  AFMC/CC  should  chair  the  Council.  In  addition,  the  ATC  should  ensure 
effective  linkage  between  concepts  for  system  development  and  the  S&T  program.  The 
operational  MAJCOMs  provide  warfighter  insights  as  well  as  Program  Objective  Memorandum 
advocacy.  The  ATC  will  include  SAF/AQ  representation,  typically  through  Program  Executive 
Officers  or  Program  Managers. 

The  Development  Planning  Activity  should  provide  staff  support  to  the  ATC.  Since  the 
development  planning  process  begins  early  in  the  system  life  cycle,  the  ATC  is  able  to 
expeditiously  identify  potential  resource  or  other  issues  for  transition.  The  development  planner 
provides  the  baseline  transition  plan  for  the  ATC  and  updates  it  as  required  due  to  changes  in 
requirements  or  technology  evolution. 
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Portfolio  Characterization 

Findings 


■  Current  characterization  of  Air  Force  S&T  program 
is  inadequate 

■  Organized  by  Laboratory  Thrust  areas  rather  than 
explicit  CFCs 

■  Better  characterization  of  Air  Force  S&T  would 
enable  budget  trade-offs  among 

■  Programs  within  the  S&T  budget 

■  S&T  and  other  programs 

■  AFRL  and  external  (for  example,  DARPA,  industry) 
funding 


10/13/00 


Integrity  -  Service  ■  Excellence 


The  Air  Force  S&T  program  includes  a  portfolio  of  projects  organized  by  six  Integrating 
Technology  Thrusts  (H  i  s):  Space  Superiority,  Precision  Strike,  Information  Dominance, 
Aircraft  Sustainment,  Aircraft  Protection,  Agile  Combat  Support.  Although  an  attempt  was 
made  to  relate  the  thrusts  to  the  Air  Force  core  competencies  and,  by  inference,  the  CFCs 
(although  they  are  not  mentioned  explicitly),  the  direct  support  of  the  Air  Force  core 
competencies  is  not  clear.  Some  ITTs  have  similar  titles  to  the  core  competencies,  but  others  do 
not.  The  panel  does  not  feel  that  this  structure  provides  the  necessary  understanding  or  visibility 
into  the  S&T  program. 

An  alternate  S&T  portfolio  characterization,  directly  aligned  with  Air  Force  core  competencies 
and  the  CFCs  associated  with  each  core  competency  described  in  Volume  3  of  the  Air  Force 
Strategic  Plan,  would  enable  the  contribution  of  S&T  to  the  Air  Force  vision  to  be  established, 
quantified,  and  managed.  This  would  provide  information  needed  to  make  informed  decisions 
on  how  to  trade  off  programs  within  the  S&T  budget,  data  to  judge  the  contribution  of  external 
partners’  resources  to  Air  Force  needs,  and  insight  that  can  be  used  by  Air  Force  leadership  to 
trade  off  S&T  investment  with  other  investments. 
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■  Provide  executive  level  visibility  into  entire  S&T  portfolio 

■  Provide  linkage  to  strategic  CFCs  (Air  Force  Strategic  Plan 
Volume  3)  and  CFGs 

■  Portray  the  following  for  each  portfolio  component 

■  %  contribution  to  CFG 

■  “Health”  (green,  yellow,  red) 

■  Funding  composition  (Air  Force  and  non-Air 
Force) 

■  Assess  status  of  required  technical 
competencies 

■  Show  impact  of  S&T  budget  adjustments  (cuts  or  adds)  in 
context  of  entire  portfolio  to  provide  the  ability  to  trade  against 
other  Air  Force  needs 


10/13/00 


Integrity  ■  Service  -  Excellence 


30 


The  characterization  of  the  S&T  portfolio  must  clearly  show  how  the  S&T  investment  enables 
the  CFCs  and  CFGs  (an  example  portfolio  characterization  tool  is  described  in  detail  in 
Appendix  A).  The  quantified  contribution  of  each  project  supporting  a  CFC  or  CFG  must  be 
determined.  The  technical  contributions  might  come  partly  from  Air  Force  S&T  and  partly  from 
other  sources,  such  as  other  government  agencies,  industry,  international  cooperation,  or  use  of 
commercial  technology.  The  characterization  would  display  the  contribution  from  all  partners. 

It  would  also  provide  an  assessment  of  the  health  and  relevance  of  technical  competencies  within 
the  laboratories  needed  to  support  the  CFCs  and  CFGs. 

The  characterization  tool  described  in  Appendix  A  represents  the  current  status  (budget, 
technical  progress,  cost,  and  schedule)  of  all  the  S&T  projects.  The  characterization  tool, 
supported  by  analysis,  will  help  define  the  impact  on  CFCs  and  CFGs  and  the  changes  in  budget, 
performance,  cost,  or  schedule. 

Viewed  as  a  total  across  all  CFCs,  the  integrated  S&T  portfolio  will  show  the  impact  of  S&T 
investment  on  the  overall  Air  Force  vision  and  will  provide  data  for  informed  decisions 
regarding  allocation  of  resources  in  the  S&T  program,  as  well  as  reallocation  (plus -ups  or  cuts) 
between  the  S&T  accounts  and  other  accounts  in  the  Planning,  Programming,  and  Budgeting 
System  process. 
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■  Implement  a  characterization  tool  to  provide 
executive -level  visibility  of  the  entire  S&T  program 
and  the  value  of  its  elements  to  support  CFCs  and 
CFGs  (AFRL/CC) 


Integrity  -  Service  -  Excellence 


The  SAB  recommends  that  the  AFRL/CC  implement  a  characterization  tool  to  provide 
executive -level  visibility  into  the  entire  S&T  program  and  the  value  of  its  elements  to  support  the 
Air  Force  CFCs  and  associated  CFGs. 
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Leveraging  Partnerships 
Slide  33:  Findings 
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■  AFRL  has  always  been  incentivized  to  partner  and  is 
doing  it  well 

■  Many  leveraged  partnerships  have  been  established 

■  Considerable  savings  have  been  realized 

■  Reduced  S&T  funding  has  reduced  the  opportunity 

■  Partner  must  be  able  to  project  reasonable  return  on 
investment  (satisfy  the  business  case) 

■  There  are  risks 

■  Government  partners  can  reprioritize  their  funding  support 

■  Industry  partners  can  change  their  IR&D  plans 

■  With  exceptions,  financial  benefits  are  not  visible 
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The  Integrated  High  Performance  Turbine  Engine  Technology  (IHPTET)  program  is  a  good 
example  of  a  leveraged  partnership.  “Partnership”  in  this  context  means  two  or  more 
organizational  entities  that  pool  their  resources  to  achieve  a  common  objective.  The  IHPTET 
program  involves  multiple  organizational  entities  (the  Army,  Navy,  Air  Force,  National 
Aeronautics  and  Space  Administration  [NASA],  General  Electric,  Pratt  &  Whitney,  and  others). 
The  principal  objectives  are  to  double  the  thrust-  to- weight  ratio,  increase  fuel  efficiency,  and 
reduce  cost  turbofan  jet  engines.  Industry  is  contributing  about  50  percent  to  the  total  effort. 

In  general,  AFRL  has  done  a  good  job  of  partnering  and  there  are  many  examples,  IHPTET 
being  just  one.  Although  they  can  always  do  better,  partnering  only  works  when  all  partners  can 
justify  the  investment.  Industry  must  be  able  to  project  a  reasonable  financial  return.  This 
generally  means  there  are  broader  applications  for  the  technology  than  DoD.  In  the  case  of 
IHPTET,  the  industrial  partners  are  clearly  looking  at  the  opportunities  in  the  commercial  airline 
marketplace.  The  Air  Force  must  justify  the  need  on  the  basis  of  warfighter  needs.  Only  when 
the  needs  of  the  warfighter  sufficiently  overlap  the  needs  of  industry  (financial  performance)  are 
the  conditions  ripe  for  a  partnership. 

It  appears  that  AFRL  is  not  getting  much  credit  for  its  partnership  efforts  to  date,  due  to  lack  of 
visibility.  The  portfolio  characterization  tool  will  provide  more  insight. 
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■  Use  portfolio  characterization  tool  to  gain  better 
insight  (AFRL/CC) 
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The  portfolio  characterization  tool  provides  visibility  into  funding  contributions  by  funding 
sources  such  as  industry  or  a  government  agency.  Funding  source  detail  is  down  to  the  levels  of 
aggregated  projects  and  technology  competency  and  is  linked  to  CFGs  and  CFCs.  Data  can  be 
aggregated  to  provide  an  executive  summary.  Trends  can  be  tracked  to  determine  whether 
partnership  leveraging  is  improving  or  deteriorating. 
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Leveraging  Commercial  Technology 
Slide  35:  Findings 
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U.S.  AIR  FORCE 


Findings 


■  AFRL  appears  to  be  coordinating  its  S&T 
investments  with  other  government  agencies 

■  However,  it  is  not  doing  as  well  in  tracking  and 
evaluating  commercial  technologies  in  some  areas 

■  Best-practice  companies  devote  S&T  resources  to 
continuously  tracking  and  evaluating  relevant 
commercial  technologies 
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AFRL  goes  to  considerable  lengths  to  coordinate  S&T  investments  with  other  government 
agencies.  An  example  is  the  Space  Technology  Alliance.  The  alliance  is  chaired  by  the  AFRL 
Space  Vehicle  Director,  and  membership  consists  of  almost  every  governmental  entity  that  is 
investing  in  space. 

AFRL  is  not  taking  sufficient  advantage  of  commercial  technologies  and  products,  as  evidenced 
by  a  major  finding  in  the  last  S&T  Quality  Review.  Companies  with  best  practices  have 
redefined  the  role  of  their  R&D  organizations.  Some  portion  of  their  organizations  and  resources 
is  dedicated  to  continuously  tracking,  evaluating,  and  adapting  relevant  commercial  technologies 
and  products,  emphasizing  fast-paced  technologies  such  as  computers,  software,  electronics,  and 
communications.  Industry  has  learned  that  evaluating  commercial  off-the-shelf  products  is  as 
important  as  keeping  track  of  what  is  happening.  Hands-on  testing  reveals  far  more  than  do 
brochures  and  data  sheets.  Those  companies  that  learn  to  leverage  and  take  advantage  of  these 
commercial  technologies  and  products  can  achieve  both  improved  performance  and  reduced  cost. 
This  improvement  can  mean  a  decided  competitive  advantage  in  the  marketplace. 
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■  Increase  emphasis  on  tracking  and  acquiring 
commercial  technology  (AFRL/CC) 

■  Incentivize  buy  before  make  behavior  (AFRL/CC) 

■  Provide  independent  assessment  via  SAB  Quality 
Review  (SAF/AQ) 
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Industry  has  learned  that  “make  or  buy”  decisions  are  best  made  when  a  justification  or 
recommendation  is  made  to  a  higher  level  that  has  final  decision  authority  and  does  not  benefit 
by  a  “make”  decision.  In  other  words,  get  the  bias  out  of  the  decision  process.  Often, 
organizations  will  rationalize  a  “make”  decision  when  “buy”  is  more  appropriate.  This  was 
evident  in  the  recent  SAB  S&T  Quality  Review  finding.  In  the  context  of  AFRL,  “make”  means 
develop  in-house  or  fund  a  subcontractor  to  develop.  The  idea  is  to  avoid  development  if  at  all 
possible.  The  lowest  overall  cost  to  the  Government  is  the  objective. 

The  SAB  S&T  Quality  Review  generally  looks  at  whether  the  laboratory  is  duplicating  or  can 
benefit  from  external  R&D.  The  independent  assessment  recommendation  would  require  the 
SAB  be  more  deliberate  in  this  regard. 
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Science  and  Engineering 

Workforce 

Findings 


■  The  STW-21  study  clearly  defines  issues  and 
directions  for  AFRL  workforce  strategy 

■  Some  successes  in  implementation  of  STW-21 
recommendations  (for  example,  relief  of  high-grade 
ceiling) 

■  Still  many  hurdles 

■  Personnel  initiatives  to  enhance  flexibility  in 
recruiting,  retaining,  rewarding,  and  removing 

■  Conversion  to  government-owned,  collaborator- 
assisted  (GOCA)  model 

■  Increase  of  flowthrough  of  “agile”  collaborators 
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The  1999  Science  and  Technology  Workforce  for  the  21st  Century  study  (STW-21),  chaired  by 
Air  Force  Chief  Scientist,  Dr.  Daniel  Flastings,  was  charged  by  SecAF  Peters  with  characterizing 
the  existing  Air  Force  S&T  workforce  and  determining  strategies  to  stretch  toward  an  ideal 
workforce.  In  many  ways,  this  S&T  study  is  a  logical  follow-on  to  STW-21. 

It  has  been  one  year  since  STW-21  was  briefed  to  the  Air  Force  senior  leadership.  There  have 
been  a  few  successes  in  the  implementation  of  the  recommendations.  For  example,  the  artificial 
ceiling  for  promotion  to  high  grades  in  the  laboratory  has  been  lifted.  Under  Secretary  of  the  Air 
Force  DeBattiste  has  been  very  involved  in  pushing  some  personnel  initiatives  through  the 
Office  of  Personnel  Management  and  the  Office  of  the  Secretary  of  Defense  (OSD).  However 
large,  important  recommendations  remain  with  only  partial  progress.  Personnel  initiatives  are 
needed  to  enhance  flexibility  in  recruiting,  retaining,  and  rewarding  high  performers  and 
removing  substandard  performers.  Conversion  to  the  GOCA  model,  through  an  increase  in 
number  and  flow-through  of  the  “agile  section”  of  the  S&E  workforce  with  term  and  temporary 
hires,  Intergovernmental  Personnel  Act  exchanges,  and  postdoctoral  fellowships,  would  augment 
the  Government  S&E  workforce  with  additional  collaborators  of  national  repute. 
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Aggressive  implementation  of  STW-21 

■  Provide  S&E  leaders  the  tools  to  recruit,  retain, 
develop,  and  manage  their  workforce  in  an  agile  and 
timely  manner  (SAF/US) 

■  Look  for  innovative  ways  to  bring  additional 
world-class  talent  into  the  lab  (AFRL/CC) 

■  Champion  and  provide  external  oversight  of  STW-21 
implementation  (AFMC/CC) 
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The  primary  recommendation  is  to  turn  up  the  heat  on  implementation  of  STW-21.  This  is 
particularly  critical  and  important  with  a  change  of  administration  in  AFRL.  All  three  key 
positions — the  commander,  executive  director,  and  chief  scientist — either  have  or  will  change 
shortly.  The  new  commander  of  AFRL,  primarily,  must  lead  the  adoption  of  STW-21  initiatives. 

Effective  implementation  of  STW-21  recommendations  will  be  enhanced  through  external 
oversight.  The  AFMC/CC  should  assume  the  role  of  working  with  AFRL  leadership,  helping  it 
to  overcome  bureaucratic  and  political  obstacles  and  overseeing  the  progress  of  this  effort. 
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Military  S&E 

Findings 


Graveyard  Spiral: 


■  Uniformed  technical  expertise  is  a  core  competency  that  is 
necessary  in  the  transformation  to  an  aerospace  continuum 

■  The  overall  military  S&E  capability  of  the  Air  Force  is  eroding 

■  Problems  in  accession,  retention,  and  development 


■  The  past  4  years  have 
created  a  looming 
crisis  due  to 
accessions  insufficient 
for  sustainment 

■  Retention  of  military 
S&Es  at  11  years  (39%) 
is  below  even  that  of 
pilots  (41%)  in  the  same 
year  group 


Engineering  Officers 
(62xxx) 


Source:  AFPC 


1  3  5  7  9  11  13  15  17  19  21  23  25  27 

Years  of  Service 
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Just  as  Dr.  Theodore  von  Karman  advocated  for  the  Air  Force  of  that  day,  the  transformation  of 
the  modem  Air  Force  into  the  aerospace  continuum  will  require  uniformed  technical  expertise. 
The  SAB  believes  that  this  military  technical  expertise  is  a  core  competency  and  should  be  cared 
for  and  managed  accordingly. 

There  is  considerable  evidence  that  the  overall  military  S&E  capability  of  the  Air  Force  is 
eroding  and  can  be  likened  to  the  aerodynamically  familiar  “graveyard  spiral”  from  which  a 
robust  recovery  must  be  applied  if  the  pilot  is  to  survive.  The  SAB  has  found  worsening  trends 
in  accession  and  retention  as  well  as  in  development  and  management  of  the  S&E  force.  The 
graph  shows  the  current  (as  of  1999)  inventory  of  military  engineers  versus  the  sustainment 
necessary  to  preserve  the  force.  After  the  first  4  years,  it  can  be  seen  that  the  inventory  roughly 
matches  the  sustainment  or  authorized  curves.  However,  from  the  first  4  years’  data  it  can  be 
seen  that  a  crisis  in  military  engineers  will  soon  occur,  if  it  has  not  occurred  already.  The  Air 
Force  has  been  accessing  during  those  4  years  at  slightly  over  half  the  rate  necessary  to  sustain 
the  force.  When  one  projects  that  4- year  period  into  the  later  periods  (say,  8-  to  12- year  middle 
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management  period),  with  normal  retention  there  may  be  fewer  than  50  military  engineers  in  any 
1 -year  group. 

It  is  interesting  to  note  that,  at  the  same  point  in  time  (11  years  of  commissioned  service),  the 
retention  of  military  S&Es  (39  percent)  is  even  worse  than  that  of  pilots  (41  percent).  The 
combination  of  lower  accession  rates  with  lower  retention  results  is  the  graveyard  spiral  that  is 
occurring  and  needs  to  be  stopped. 
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Military  S&E 

Findings 


■  Fills  of  advanced  academic 
degree  (AAD)  training  slots 
are  far  short  of  quotas 

■  Total  quotas  seem 
adequate;  however, 
distribution  may  be  out  of 
line  with  critical  Air  Force 
needs 


AFIT  S&E 
Quotas 

MS 

Quota 

MS 

Fills 

PhD 

Quota 

PhD 

Fills 

Aero/Astro/Mech 

32 

10 

10 

0 

Acq  Mgmt 

21 

20 

0 

0 

Comp  Sci/Engr 

22 

9 

3 

0 

Elect  Engr 

43 

21 

8 

2 

Electro  Optics 

9 

2 

1 

0 

Engr 

Phys/Nuclear 

17 

6 

8 

3 

Environmental 

12 

12 

0 

0 

Logistics  Mgmt 

21 

7 

0 

0 

Meteorology 

22 

12 

2 

0 

Ops  Anal/Rsch 

20 

13 

3 

0 

Sys  Eng/Space 
Ops 

11 

2 

0 

0 

TOTAL 

230 

128 

35 

5 

The  Air  Force  cannot  continue  as  a  high-tech  force  without  S&E  officers 
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A  further  serious  deficit  occurs  with  respect  to  the  advanced  academic  degree  education  of  S&E 
officers.  The  SAB  recommends  that  at  least  half  of  the  S&E  officers  have  technical  master’s 
degrees  and  that  15  percent  have  technical  Ph.D.s.  This  level  of  education  is  necessary  to 
provide  the  technical  leadership  needed  for  high-technology  acquisitions  occurring  within  the 
Air  Force. 

The  table  shows  the  quota  versus  fills  for  several  Air  Force  Institute  of  Technology  (AFIT) 
engineering  degree  programs  in  FY00.  The  first  observation  is  that  the  total  fills  do  not  come 
close  to  filling  the  quotas — 128  of  230  at  the  master’s  level  and  5  of  35  at  the  Ph.D.  level. 
Equally  important,  there  are  serious  distribution  deficiencies  of  the  programs  with  respect  to  Air 
Force  needs.  For  example,  electro- optics  is  critical  in  directed- energy  applications  and 
intelligence,  surveillance,  and  reconnaissance  such  as  Space- Based  Infrared  System,  the 
Airborne  Faser,  and  the  Space-Based  Faser.  Yet  there  are  no  Ph.D.  fills  and  only  two  master 
fills.  Similarly,  in  computer  science  and  computer  engineering,  which  are  key  to  command  and 
control  and  information  operation  applications,  there  are  no  Ph.D.  fills  and  less  than  half  the 
masters  needed. 
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The  process  for  generating  AAD  requirements  could  be  improved.  Not  only  are  officers  not 
necessarily  being  educated  in  the  right  fields,  but  AFIT  cannot  be  expected  to  operate  efficiently 
with  such  low  and  unstable  input. 

The  Air  Force  will  be  in  serious  trouble  technically  if  this  situation  is  allowed  to  continue.  The 
SAB  believes  that  the  Air  Force  cannot  continue  as  a  high-technology  force  without  S&E 
officers  who  have  appropriate  advanced  academic  technical  competence. 
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Recommendations 
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■  Increase  S&E  accessions  (AETC/CC  and  USAFA/CC) 

■  Align  technical  degree  quotas  with  needs,  then  fill 
them  (AF/DP) 

■  Institute  an  improved  career  management  plan  for 
S&E  officers  to  emphasize  their  value  and  visibility 
to  the  Air  Force  (S&E  Functional  Manager) 

■  Manage  the  S&E  officer  corps 
(SAF/AQ-OPR,  AFMC/CC-OCR) 

■  Note:  S&E  summit  scheduled  in  Dec  2000  to  define 
the  path  forward 

Stop  the  military  S&E  graveyard  spiral! 
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A  reasonable  first  step  is  to  initiate  a  policy  to  up  the  gain  in  S&Es  from  the  three  officer 
accession  sources  (U.S.  Air  Force  Academy,  Reserve  Officer  Training  Corps,  and  Officer 
Training  School),  with  strategic  vectoring  of  graduates  into  critical  fields. 

The  quotas  for  advanced  technical  degrees  should  be  increased,  but  even  more  important,  the 
quotas  need  to  be  filled.  The  SAB  is  concerned  about  the  apparent  lack  of  incentive  for  an 
officer  to  attain  an  advanced  technical  degree.  The  hurdles  (for  example,  releasing  eligible 
officers  for  school  because  of  increased  ops  tempo)  should  be  identified  and  removed  so  that 
officers  have  an  incentive  to  pursue  an  advanced  degree  and  see  it  as  career- enhancing. 

A  new  revised  career  management  plan  should  be  instituted  to  enhance  the  career  opportunities 
and  progression  for  S&E  officers.  This  will  require  senior- level  attention  from  SAF/AQ. 
Furthermore,  the  S&E  officer  should  feel  that  their  careers  are  the  concern  of  at  least  one  very 
senior  Air  Force  officer,  as  is  the  case  for  rated  personnel. 

The  SAB  feels  that  these  recommendations  are  the  minimum  corrections  to  stop  the  graveyard 
spiral.  We  feel  that  they  must  be  applied  soon  and  persistently  to  be  effective. 


49 


Slide  42:  Conclusion 


U.S.  AIR  FORCE 


Conclusion 


■  Summary  Recommendations 

■  Terms  of  Reference  Compliance 
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Slide  43:  Summary  Recommendations  for  SecAF  and  CSAF 


Summary  Recommendations 

***  for  SecAF  and  CSAF 
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■  Set  specific  CFGs  that  provide  a  basis  for  key  system 
and  operational  concepts  and  S&T  planning 

■  Linked  to  CFCs  in  Vol  3,  Air  Force  Strategic  Plan 

■  Updated  annually 

■  Hold  AFRL/CC  accountable  for  formulating  and 
executing  an  S&T  plan  that  achieves  CFGs  and  other 
warfighter  requirements 

■  Preside  over  the  annual  review  of  S&T  plan  and 
execution  of  previous  year’s  plan 

■  S&T  portfolio  presentation  should  link  entire  S&T 
investment  to  CFCs  and  emphasize  major  technology 
demos  and  attainment  of  CFGs  and  other  key  warfighter 
requirements 


Integrity  -  Service  -  Excellence 


51 


Slide  44:  Summary  Recommendations  for  SecAF  and  CSAF  (continued) 


i  i  Summary  Recommendations 

(Confd) 

s.  air  force _ for  SecAF  and  CSAF 

■  Create  program  element  to  reenergize  development 
planning 

■  Hold  AFMC/CC  accountable  to  ensure  system-of- 
systems  focus  across  product  lines 

■  Formulate  systems  and  operational  concepts  that 
address  CFGs  and  other  warfighter  requirements 

■  Provide  basis  for  prioritization  of  S&T  plan  based  on 
rigorous  system  trade  studies 

■  Direct  increased  emphasis  on  accession,  retention, 
and  development  of  S&E  officers 

■  Note:  S&E  summit  scheduled  in  Dec  2000  to  define 
path  forward 
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Slide  45:  Summary  Recommendation  for  Air  Force  Service  Acquisition  Executive  (SAE) 


Summary  Recommendation 

***  for  Air  Force  SAE 
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■  Hold  AFRL/CC  accountable  for  executing  S&T  plans  to 
provide  needed  technical  performance  on  agreed 
schedule  for  agreed  cost 
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Slide  46:  Summary  Recommendations  for  AFMC/CC 


Summary  Recommendations 

***  for  AFMC/CC 
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■  Lead  implementation  of  revitalized  development 
planning,  integrating  it  across  product  centers, 
and  use  it  to  establish  priorities  for  the  S&T  plan 

■  Advocate  the  S&T  program  and  budget  as 
represented  in  the  S&T  plan  into  new  Air  Force 
Resource  Allocation  Process 

■  Chair  ATC  when  competing  concepts  involve 
multiple  product  centers 

■  Play  increased  role  in  S&E  officer  development 
and  retention 
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Slide  47:  Summary  Recommendations  for  AFRL/CC 


Summary  Recommendations 

***  for  AFRL/CC 

U.S.  AIR  FORCE _ 

■  Use  trade  studies  from  development  planning  to 
focus  and  prioritize  S&T  investments  to  achieve 
CFGs  and  meet  other  warfighter  requirements 

■  Characterize  the  value  of  the  entire  S&T  program  in 
warfighter  terms  and  present  the  characterization  to 
Air  Force  leadership  annually 

■  Hold  program  managers  accountable  for  cost, 
schedule,  and  performance  for  each  CFG  and  other 
key  projects 

■  Increase  emphasis  on  tracking  and  acquiring 
commercial  technology — incentivize  buy  before  make 
behavior 
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Slide  48:  Terms  of  Reference  Compliance 


Terms  of  Reference 
usJLcE  Compliance 

Terms  of  Reference  Item 

Compliance  Action 

Tie  S&T  investment  strategy  to  Air  Force 
long-  range  plan  and  vision 

■  Leadership  sets  CFGs  to  drive  concepts  to 
drive  S&T  plan 

Take  into  account  the  investment  of 
others  in  Air  Force  S&T  strategy,  but 
recognize  risk;  partnership  with  industry 

■  Partnerships 

■  Leveraging  external  technology 

Recommend  industry  tools  (practices)  for 
Air  Force  to  use  in  determining  S&T 
investment 

■  Defined  tool  for  characterizing  S&T  portfolio 
at  executive  level  (valuing  and  prioritizing 

S&T) 

Develop  augmenting  strategies  for  work 
force  management,  risk  assessment,  and 
hedges 

■  Recommended  actions  to  deal  with  Air 

Force  S&E  officer  and  civilian  workforce 
issues 

■  S&T  portfolio  characterization  tool  defines 
dependencies  on  non-Air  Force  funding  sources 

Recommend  a  level  for  Air  Force  S&T 
topline 

■  Process  for  setting  CFGs  and  S&T  portfolio 
characterization  tool  provide  basis  for 
setting  S&T  topline  in  light  of  other  Air 

Force  needs 

■  Sanity  checks  provided  by  previous  studies 

Integrity  -  Service  -  Excellence 

This  chart  correlates  the  various  findings  and  recommendations  of  the  study  to  the  Terms  of 
Reference. 

In  the  first  item,  the  study  recommends  a  process  through  which  leadership  goals  will  ultimately 
be  reflected  in  the  S&T  plan.  In  the  second  item,  specific  recommendations  are  made  in  the 
areas  of  partnerships  and  leveraging  of  external  technology.  Item  three  was  addressed  by 
defining  a  tool  for  executive- level  characterization  of  the  S&T  portfolio.  This  tool,  once  created, 
will  provide  decision  makers  with  information  appropriate  for  valuing  and  prioritizing  the  S&T 
portfolio. 

Specific  recommendations  concerning  both  the  civilian  and  military  workforces  are  made  in 
compliance  with  item  four.  The  portfolio  characterization  tool  described  in  item  three  also 
addresses  this  item. 

The  final  item  was  not  addressed  directly.  Rather,  a  process  is  described  that  will  allow  setting 
the  topline  budget. 
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Appendix  A:  Conceptual  Model  for  S&T  Characterization 


Slide  49:  Example  S&T  Portfolio  Characterization  Tool 
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Slide  50:  Conceptual  Model 
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A  conceptual  S&T  portfolio  characterization  model  is  depicted  in  this  chart.  At  the  aggregate 
level,  the  entire  S&T  program  is  linked  to  the  CFCs.  As  can  be  seen,  the  CFGs  are  linked  to  the 
appropriate  CFCs,  but  not  all  CFCs  have  CFGs.  Also,  several  CFGs  may  be  associated  with  one 


CFC. 


There  are  essentially  four  elements  to  this  model.  First,  there  are  specific  projects  (primarily  6.3, 
but  with  some  complement  of  6.2  funding)  that  focus  on  achieving  specific  CFGs.  There  are 
also  projects  associated  with  Technology  Competencies,  primarily  6.2- funded  technology  areas 
needed  to  support  a  broader  range  of  Air  Force  missions,  but  that  support  the  achievement  of  one 
or  more  CFGs.  The  third  group  of  projects  are  those  linked  to  achievement  of  the  broader  CFC, 
but  not  a  specific  CFG.  These  will  likely  be  both  6.2-  and  6.3-funded  projects.  Finally,  there  is 
the  basic  research  program  (6.1),  which  provides  the  foundation  for  future  technology 
development,  enabling  the  next  set  of  CFGs  and,  thus,  supports  all  CFC  statements. 

The  SAB  envisions  that  aggregated  projects  will  be  hyperlinked  to  this  summary  document  in  a 
manner  that  allows  more  detailed  information  to  be  viewed  if  necessary  to  understand  the  top- 
level  information  displayed. 

The  important  point  is  that,  once  developed,  this  tool  will  provide  the  Air  Force  with  a  means  of 
easily  portraying  connectivity  between  the  S&T  program  and  the  future  Air  Force. 
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Slide  51:  Assessing  the  Value  of  S&T 
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This  tool  has  additional  benefits.  There  are  a  number  of  metrics,  such  as  the  two  depicted  here, 
that  can  be  captured  and  displayed.  In  the  first  example,  the  bars  represent  individual 
aggregations  of  projects  and  the  agency  (or  industry)  providing  the  funds.  The  individual  bars 
are  color- coded  in  proportion  to  the  funding  provided.  For  example,  DARPA  is  providing 
approximately  60  percent  of  the  funds  needed  to  execute  the  first  project  under  CFGi .  This 
provides  a  ready  mechanism  to  visually  depict  the  degree  of  leveraging  within  AFRL  and  track  it 
over  time.  It  also  can  be  used  as  an  indicator  of  the  funding  risk  tied  to  achieving  the  CFGs  and 
CFCs. 


The  second  example  shows  a  method  of  depicting  the  health  of  projects.  Earlier  in  this  report, 
the  SAB  recommended  that  CFGs  have  specific  cost,  schedule,  and  performance  objectives 
much  like  an  acquisition  program,  and  that  the  AFRL/CC  be  held  accountable  for  achieving 
those  objectives.  Health  can  also  be  depicted  for  Technical  Competencies  with  appropriate 
metrics.  This  depiction,  when  aggregated  for  all  CFCs,  provides  an  executive -level  means  to  do 
exactly  that.  It  also  provides  cues  to  problem  areas  that  can  be  explored  because  each  cell  is 
hyperlinked  to  more  detailed  information. 

This  chart  only  displays  two  metrics,  but  others  can  be  easily  added  as  needed.  The  SAB 
believes  that  this  tool  provides  a  mechanism  to  track  and  display  (at  an  executive  level,  with 
hyperlinked  backup  information)  any  metric  needed  to  assess  the  execution  and  health  of  the 
S&T  program. 
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Slide  52:  Assessing  the  Impact  of  Budget  Adjustments 


Characterization 

Assessing  the  Impact  of  Budget  Adjustments 

U.S.  AIR  FORCE  3 


Before 


After 


. _ 

cfc8 


cfc9 


_ 

cfc8 


\ 

cfc9 


10/13/00 


Integrity  ■  Service  -  Excellence 


52 


This  characterization  tool  can  also  be  used  to  understand  the  impact  of  budgetary  adjustments. 

In  the  above  example,  moving  technology  development  funds  from  one  CFC  statement  to 
another  is  being  considered.  This  tool  allows  for  executive- level  depiction  of  the  impact  of  that 
proposal  and  allows  senior  Air  Force  leadership  to  better  assess  the  impact.  Because  each  cell  is 
hyperlinked  to  more  detailed  information,  specific  areas  can  be  investigated  if  necessary  to  make 
more  informed  decisions. 
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Slide  53:  Notional  Example — Targets  Under  Trees 
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To  further  understand  how  the  characterization  tool  might  be  used,  consider  the  notional  example 
of  finding  targets  under  trees.  In  this  example,  top  Air  Force  leadership  has  established  a  CFG  to 
find,  identify,  and  kill  targets  under  trees  globally  within  3  hours  of  tasking.  Noting  that  the 
health  of  one  of  the  projects,  very  high  frequency  (VHF)  change  detection  (CD),  has  been  rated 
“yellow”;  more  details  about  the  execution  of  the  CFG  projects  are  requested.  Clicking  on  the 
appropriate  CFG  displays  the  detailed  bar  chart  shown  on  the  next  slide. 


A -5 


Slide  54:  Notional  Example — Targets  Under  Trees  (continued) 
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This  chart  provides  a  brief  summary  of  the  overarching  objective  of  the  TUT  CFG.  It  also 
provides  insight  into  metrics  for  the  three  projects  developing  technology  for  TUT. 

To  successfully  achieve  the  objectives  of  this  CFG,  it  is  necessary  to  detect,  identify,  and  engage 
hostile  targets.  Through  analysis  of  existing  equipment  and  projects  in  development,  it  was 
determined  that  the  best  method  to  detect  targets  within  the  necessary  time  lines  was  through  a 
combination  of  ultra  high  frequency  (UHF)  and  VHF  synthetic  aperture  radars  (SARs)  on  a 
Global  Hawk  UAV.  DARPA  was  already  developing  the  UHF  SAR  system,  so  AFRL 
undertook  the  development  of  the  VHF  SAR  receiver.  To  identify  targets,  it  was  necessary  to 
fuse  this  data  with  data  from  other  intelligence  sources.  Finally,  a  new  fuze  required 
development  in  order  to  engage  and  destroy  hard  targets,  such  as  tanks,  using  only  one  bomb. 
Thus,  the  VHF  CD  project,  the  Sensor  Fusion  project,  and  the  Height  of  Burst  (HOB)  Fuze  form 
the  core  projects  for  TUT. 

From  the  first  metric,  it  can  be  seen  that  more  than  half  of  the  funding  for  VHF  CD  and  Sensor 
Fusion  is  provided  by  DARPA,  while  the  Air  Force  is  funding  the  entire  HOB  Fuze  project. 

Thus,  the  Air  Force  is  heavily  leveraging  the  investment  of  DARPA  to  satisfy  this  CFG. 


A -6 


The  second  metric  shows  that,  between  VHF  CD  and  Sensor  Fusion,  if  successful,  about 
80  percent  of  the  desired  capability  is  achieved.  Thus,  this  provides  a  measure  of  the  importance 
of  each  project  to  achieving  the  CFG. 

Finally,  from  the  “Health”  metric,  which  summarizes  the  cost,  schedule,  and  performance  of 
each  project,  we  can  see  that  there  is  a  problem  with  the  VHF  CD  project.  The  hyperlink  to  this 
project  reveals  more  details. 
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Slide  55:  Notional  Example — Targets  Under  Trees — VHF  Change  Detection 
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Hyperlinking  from  the  VHF  CD  project  title  displays  a  summary  of  this  project.  From  this  chart, 
we  see  that  the  performance  of  the  VHF  CD  project  is  being  impacted  by  higher- than- predicted 
false  alarms  in  DARPA’s  Foliage  Penetration  (FOPEN)  Radar  program.  The  workaround 
solution  for  this  technical  difficulty,  is  to  rely  more  on  the  VHF  sensor  and  modify  the  CD 
algorithms  to  account  for  this  shift. 

This  example  illustrates  the  power  inherent  in  the  proposed  characterization  tool.  Air  Force 
leadership  can  rapidly  receive  information  about  the  S&T  program  at  the  portfolio  level,  CFC 
level,  CFG  level,  or  project  level,  if  necessary.  Multiple  metrics  can  be  tracked  concurrently. 

This  suggested  tool  provides  a  valuable  method  to  assess  the  connectivity  of  the  S&T  program  to 
the  future  Air  Force,  while  simultaneously  assessing  its  value  and  health. 
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Appendix  B 
Terms  of  Reference 


USAF  Scientific  Advisory  Board  2000  Study 
Science  &  Technology  and  the  Air  Force  Vision:  A  Critical  Partnership  and 

Strategy  for  the  Future 

BACKGROUND:  In  1995  as  part  of  New  World  Vistas,  the  SAB  reviewed,  through  personal 
interviews  with  leaders  of  aerospace  industry,  R&D  investments  with  emphasis  on  long-term. 

Their  findings  are  most  dramatically  captured  in  one  prominent  CEO’s  statement:  I  am 
embarrassed  to  say  that  we  no  longer  do  long-term  R&D. 

Industry  is  driven  by  earnings  to  share  holders,  which  forces  their  investments  to  be  largely  near- 
term  and  R&D  becomes  R&D — little  research  and  mostly  development.  Whether  industry  does 
true,  long-term  R&D  or  not,  it  does  invest  in  research.  Companies  have  developed  sophisticated 
tools  for  guiding  these  investments  and  developing  strategic  partnerships  with  universities  and 
industry. 

The  AFRL  was  created  in  1997  with  one  of  its  goals  that  of  better  tying  of  S&T  investments  to 
Air  Force  needs  and  its  future.  Portfolios  were  developed  for  each  of  the  Directorates  mindful  of 
investments  in  related  areas  in  other  Services,  DARPA,  OSD,  and  defense  agencies.  Investments 
by  industry  were  also  considered,  but  unevenly  across  the  Directorates.  Since  its  creation,  AFRL 
has  divested  either  largely  or  completely  in  some  areas  based  on  the  above  analysis  and  budget 
pressures.  The  recent  S&T  topline  reductions  have  weakened  what  was  a  well-balanced 
portfolio  development  5  years  ago  in  several  critical  areas. 

Study  Products:  Briefing  to  SecAF  and  CSAF  in  October  2000.  Publish  report  in  December 
2000. 

Charter:  This  study,  guided  by  the  Air  Force  Long  Range  Plans,  Air  Force  Vision,  and  U.S. 
National  and  Military  Strategies,  will  develop  an  Air  Force  strategy  for  dealing  with  these 
realities  by: 

1.  Developing  a  technology  investment  strategy  informed  by  those  documents  and  the  best  technical 
minds  regarding  Air  Force  needs  and  technology  possibilities/opportunities. 

2.  Reviewing  industry  investments  to  determine  level  of  effort  in  Air  Force  S&T  being  mindful  of  the 
potential  divestiture  by  industry  in  any  given  area  (how  long  can  the  Air  Force  depend  on  an  industry 
source — affects  work  force  and  minimum  level  of  effort  on  a  topic,  for  example,  heavy  DARPA 
funding  in  information  technology). 

3.  Reviewing  the  tools  industry  uses  to  determine  where  to  invest  and  form  strategic  partnerships  and 
recommend  tools  that  can  be  used  by  AFRL. 

4.  Developing  and  recommending  strategies  for  S&T  investment  at  all  levels  to  include  industry 
partnerships,  work  force  implications,  risk  assessment  and  hedges. 

5.  From  the  above,  recommending  a  level  for  the  Air  Force  S&T  topline. 

The  study  will  review  all  recent  reports  related  to  this  subject  (for  example,  Defense  Science 
Boai'd)  to  avoid  duplication.  This  study  will  answer  a  Fall  ’99  CORONA  tasker  deriving  from 
the  S&T  Funding  Strategy  presentation  there.  It  grows  from  a  continuing  SAB  concern  about 
and  partnership  with  AFRL  regarding  the  best  use  of  Air  Force  S&T  resources  and  leveraging 
opportunities. 
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Appendix  C 
Study  Team 


Study  Chairman 

Dr.  William  F.  Ballhaus,  Jr. 

SAB  Military  Director 

Lt  Gen  Stephen  B.  Plummer 

General  Officer  Participants 

Brig  Gen  Paul  Nielsen 

Maj  Gen  Richard  Paul,  USAF  (Ret) 

SAB  Executive  Director 

Col  Gregory  H.  Bishop 

SAB  Study  Executive  Officer 

Ms.  Melody  Darby,  AFRL/XPPR 
Lt  Col  Paul  A.  Schubert,  AF/SB 

SAB  Study  Technical  Writer 

Maj  Larry  Merkle,  USAFA 

Team  Leaders 

CEO  Guidance  and  Participation  Team:  Lt  Gen  George  K.  Muellner,  USAF,  (Ret) 
Linkages  and  Transitions  Team:  Lt  Gen  Malcolm  R.  O'Neill,  USA,  (Ret) 

Funding  Guidelines  and  Leveraging  External  Sources:  Dr.  Antonio  F.  Pensa 
Effective  Planning  Process  Team:  Dr.  Valerie  J.  Gawron 
Other  Factors:  Dr.  Daniel  E.  Hastings 
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Study  Members 


Dr.  William  F.  Ballhaus,  Jr.,  Chair 

Corporate  Vice  President,  Engineering  and  Technology 

Lockheed  Martin  Corporation 

Mr.  Jeffrey  E.  Grant,  Deputy  Chair 
Private  Consultant 

Dr.  Robert  R.  Barthelemy 
Director  of  Education  and  Training 
Universal  Technology  Corporation 

Col  William  A.  Byme,  USAF  (Ret) 

Director,  Albuquerque  Operations 
Universal  Technology  Corporation 

Mr.  Art  Chester 
President 

HRL  Laboratories 

Mrs.  Natalie  W.  Crawford 

Vice  President  and  Director,  Project  AIR  FORCE 

RAND 

Col  Donald  R.  Erbschloe 
HQ  USAF/ST 

Dr.  Janet  Fender 

Chief  Scientist,  Space  Vehicles  Directorate 
Air  Force  Research  Laboratory 

Dr.  Matthew  W.  Ganz 

Vice  President  for  Programs 

The  Charles  Stark  Draper  Laboratory,  Inc. 

Dr.  Valerie  J.  Gawron 
Principal  Human  Factors  Engineer 
Veridian  Engineering 

Dr.  Daniel  E.  Hastings 
Professor 

Massachusetts  Institute  of  Technology 
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Lt  Col  Walter  E.  Hammond,  USAFR,  Ph.D. 

Special  Projects  Engineer 
Hernandez  Engineering,  Inc. 

Prof.  James  A.  Hendler 
IPA  to  DARPA 
University  of  Maryland 

Maj  Gen  Donald  L.  Lamberson,  USAF  (Ret),  Ph.D. 
Private  Consultant 

Dr.  Louis  S.  Metzger 
Chief  Scientist 
AF/ST 

Lt  Gen  George  K.  Muellner,  USAF  (Ret) 

Vice  President/General  Manager,  Phantom  Works 
The  Boeing  Company 

Lt  Gen  Malcolm  O’Neill,  USAF  (Ret) 

Vice  President,  Air  and  Missile  Defense 
Lockheed  Martin  Corporation 

Dr.  Antonio  F.  Pensa 
Head  of  Aerospace  Division 
M.I.T.  Lincoln  Laboratory 

Maj  Gen  Robert  R.  Rankine,  Jr.,  USAF  (Ret),  Ph.D. 
Vice  President,  Government  Requirements 
Hughes  Space  &  Communications  Co. 

Dr.  Elsa  Reichmanis 

Head,  Polymer  and  Organic  Materials  Research 
Bell  Laboratories,  Lucent  Technologies 

Dr.  G.  Keith  Richey 

Senior  Technology  Consultant 

Universal  Technology  Corporation 

Dr.  William  B.  Rouse 
Chief  Executive  Officer 
Enterprise  Support  Systems 
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Dr.  Robert  W.  Selden 
Private  Consultant 

Dr.  James  C.  Williams 
Honda  Professor 
The  Ohio  State  University 

Dr.  Gerold  Yonas 

Vice  President  and  Principal  Scientist 
Sandia  National  Laboratories 
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Appendix  D 

Acronyms  and  Abbreviations 


AAC 

AAD 

AETC 

AF/DP 

AFIT 

AFMC 

AFPC 

AFRL 

AMSAA 

ARL 

ASC 

ATC 

BMDO 

CC 

CD 

CEO 

CFC 

CFG 

CONOPS 

CSAF 

CTO 

DARPA 

DERA 

DoD 

DSB 

ESC 

FOPEN 

GOCA 

GPS 

HOB 

IHPTET 

IR&D 

ITT 

MAJCOM 

MET 

NASA 

OCR 

OPR 

OSD 

R&D 

RDEC 

S&E 


Air  Armament  Center 

Advanced  Academic  Degree 

Air  Education  and  Training  Command 

Air  Force  Deputy  Chief  of  Staff,  Personnel 

Air  Force  Institute  of  Technology 

Air  Force  Materiel  Command 

Air  Force  Personnel  Center 

Air  Force  Research  Faboratory 

Army  Materiel  Systems  Analysis  Activity 

Army  Research  Faboratory 

Aeronautics  Systems  Center 

Applied  Technology  Council 

Ballistic  Missile  Defense  Organization 

Commander 

Change  Detection 

Chief  Executive  Officer 

Critical  Future  Capability 

Critical  Future  Goal 

Concept  of  Operations 

Chief  of  Staff  of  the  Air  Force 

Chief  Technical  Officer 

Defense  Advanced  Research  Projects  Agency 
Defense  Evaluation  and  Research  Agency 
Department  of  Defense 
Defense  Science  Board 
Electronic  Systems  Center 
Foliage  Penetration 

Government-  Owner,  Collaborator-  Assisted 
Global  Positioning  System 
Height  of  Burst 

Integrated  High  Performance  Turbine  Engine 
Technology 

Independent  Research  and  Development 
Integrating  Technology  Thrust 
Major  Command 

Massachusetts  Institute  of  Technology 
National  Aeronautics  and  Space  Administration 
Office  of  Coordinating  Responsibility 
Office  of  Primary  Responsibility 
Office  of  the  Secretary  of  Defense 
Research  and  Development 
Research,  Development,  and  Engineering  Center 
Science  and  Engineering 
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S&T 

Science  and  Technology 

SAB 

Air  Force  Scientific  Advisory  Board 

SAE 

Service  Acquisition  Executive 

SAF/AQ 

Assistant  Secretary  of  the  Air  Force,  Acquisition 

SAF/US 

Under  Secretary  of  the  Air  Force 

SAR 

Synthetic  Aperture  Radar 

SecAF 

Secretary  of  the  Air  Force 

SMC 

Space  and  Missile  Systems  Center 

SPO 

System  Program  Office 

STW-21 

Science  and  Technology  Workforce  for  the  21st 
Century  Study 

TARDEC 

Tank  Automotive  Research,  Development,  and 
Engineering  Center 

TUT 

Targets  Under  Trees 

UAV 

Unmanned  Aerial  Vehicle 

UHF 

Ultra  High  Frequency 

USAFA 

U.S.  Air  Force  Academy 

VHF 

Very  High  Frequency 
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Organizations  Consulted 


Air  Force 

ABL  SPO 

ACC/CC 

AF/CV 

AF/ST 

AF/XP 

AFMC/DRX 

AFOSRAFRL/CA 

AFRL/CC 

AFRL/MN 

AFSPC/CS 

AIA/CAASC/CC 

ASC/CD 

ASD/311HSW 

AW  ACS  SPO 

ESC/CCSAF/AQ 

SAF/AQR 

SB  IRS  SPOSMC/CC 


Industry 

3M 

Boeing 

DuPont 

General  Electric 

Hughes 

IBM 

Lockheed  Martin 

Lucent 

Microsoft 

Nokia 

Sun 


E-l 


Other  Government  Agencies 

AMSAA 
ARDEC 
ARLCERDEC 
DARPA 
DAS  (STE) 

DASA  (RT) 

DERA 

DUSD  (S&T) 

ERDEC 

MCCDC 

MCSC 

MCWL 

MICOM  RDEC 

NATO/RT 

NIST 

NRDEC 

NRL 

ONR 

SOCOMTARDEC 
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Headquarters  Air  Force 


SAF/OS 

AF/CC 

AF/CV 

AF/CVA 

AF/HO 

AF/ST 

AF/SC 

AF/SG 

AF/SF 

AF/TE 


Secretary  of  the  Air  Force 
Chief  of  Staff 
Vice  Chief  of  Staff 
Assistant  Vice  Chief  of  Staff 
Historian 
Chief  Scientist 

Communications  and  Information 
Surgeon  General 
Security  Forces 
Test  and  Evaluation 


Assistant  Secretary  of  the  Air  Force 


SAF/AQ 

SAF/AQ 

SAF/AQI 

SAF/AQL 

SAF/AQP 

SAF/AQQ 

SAF/AQR 

SAF/AQS 

SAF/AQX 

SAF/MI 

SAF/SN 

SAF/SX 


Assistant  Secretary  for  Acquisition 

Military  Director,  USAF  Scientific  Advisory  Board 

Information  Dominance 

Special  Programs 

Global  Power 

Global  Reach 

Science,  Technology  and  Engineering 
Space  and  Nuclear  Deterrence 
Management  Policy  and  Program  Integration 

Assistant  Secretary  (Manpower,  Reserve  Affairs,  Installations  &  Environment) 
Assistant  Secretary  (Space) 

Deputy  Assistant  Secretary  (Space  Plans  and  Policy) 


Deputy  Chief  of  Staff,  Air  and  Space  Operations 


AF/XO 

AF/XOC 

AF/XOI 

AF/XOJ 

AF/XOO 

AF/XOR 


DCS,  Air  and  Space  Operations 
Command  and  Control 

Intelligence,  Surveillance,  and  Reconnaissance 
Joint  Matters 
Operations  and  Training 
Operational  Requirements 


Deputy  Chief  of  Staff,  Installations  and  Logistics 

AF/IL  DCS,  Installations  and  Logistics 

AF/ILX  Plans  and  Integration 


Deputy  Chief  of  Staff,  Plans  and  Programs 


AF/XP 

AF/XPI 

AF/XPM 

AF/XPP 

AF/XPX 

AF/XPY 


DCS,  Plans  and  Programs 
Information  and  Systems 
Manpower,  Organization  and  Quality 
Programs 

Strategic  Planning 
Analysis 
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Deputy  Chief  of  Staff,  Personnel 

AF/DP 

DCS,  Personnel 

Office  of  the  Secretary  of  Defense 

USD  (A&T) 

Under  Secretary  for  Acquisition  and  Technology 

USD  (A&T)/DSB 

Defense  Science  Board 

DARPA 

Defense  Advanced  Research  Projects  Agency 

DIA 

Defense  Intelligence  Agency 

DISA 

Defense  Information  Systems  Agency 

BMDO 

Ballistic  Missile  Defense  Organization 

Other  Air  Force  Organizations 

AC2ISRC 

Aerospace  Command,  Control,  Intelligence,  Surveillance,  and 
Reconnaissance  Center 

ACC 

Air  Combat  Command 

-  CC 

-  Commander,  Air  Combat  Command 

-  366th  Wing 

-  366th  Wing  at  Mountain  Home  Air  Force  Base 

AETC 

Air  Education  and  Training  Command 

-  AU 

-  Air  University 

AFMC 

Air  Force  Materiel  Command 

-  CC 

-  Commander,  Air  Force  Materiel  Command 

-  EN 

-  Directorate  of  Engineering  and  Technical  Management 

-  AFRL 

-  Air  Force  Research  Laboratory 

-  SMC 

-  Space  and  Missile  Systems  Center 

-  ESC 

-  Electronic  Systems  Center 

-  ASC 

-  Aeronautics  Systems  Center 

-  HSC 

-  Human  Systems  Center 

-  AFOSR 

-  Air  Force  Office  of  Scientific  Research 

AFOTEC 

Air  Force  Operational  Test  and  Evaluation  Center 

AFSAA 

Air  Force  Studies  and  Analyses  Agency 

AFSOC 

Air  Force  Special  Operations  Command 

AFSPC 

Air  Force  Space  Command 

AIA 

Air  Intelligence  Agency 

AMC 

Air  Mobility  Command 

NAIC 

National  Air  Intelligence  Center 

NGB/CF 

National  Guard  Bureau 

PACAF 

Pacific  Air  Forces 

USAFA 

U.S.  Air  Force  Academy 

USAFE 

U.S.  Air  Forces  in  Europe 

U.S.  Army 

ASB 

Army  Science  Board 

U.S.  Navy 

NRAC 

Naval  Studies  Board 

Naval  Research  Advisory  Committee 
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U.S.  Marine  Corps 

DC/S  (A)  Deputy  Chief  of  Staff  for  Aviation 


Joint  Staff 

JCS 

J2 

J3 

J4 

J5 

J6 

J7 

J8 


Office  of  the  Vice  Chairman 

Intelligence 

Operations 

Logistics 

Strategic  Plans  and  Policies 

Command,  Control,  Communications,  and  Computer  Systems 
Operational  Plans  and  Interoperability 
Force  Structure,  Resources  and  Assessment 


Other 

Aerospace  Corporation 

ANSER 

MITRE 

RAND 

Study  Participants 
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